Network communications within and between organizations transfer a variety of data and vital information. However network communications are also one of the main ways attackers get inside the security perimeter of a network. Modern network security technology deploys many methods to prevent the attackers from causing harm to the organizations and individuals. None of them are perfect, especially today in the era of Advanced Persistent Threat (APT). Modern APTs can be developed and deployed targeting very specific targets and usually such APTs can be very hard to detect and eliminate. Recording the network traffic is one of the common methodologies in the advanced network defense. It is analogous to the security camera—while not optimized for the detection of the threat, it is very useful in the elimination stage, allowing a close examination of the threat-related subset of the traffic with the purpose of determining the root cause and infected network nodes.
The network traffic is ever increasing and that creates a challenge for a traditional network recording methods. For APT sometimes taking around months to fully penetrate the organization, a typical requirement for the network data storage is at least 6 months. For a large enterprise it is usually either impossible or impractical to collect and store the resulting amount of data. Even if the APT infection happens, the traffic caused by this attack is miniscule in comparison with the amount of data the enterprise is forced to store in order to research that attack.
The current invention offers a methodology allowing significantly reducing the amount of data that needs to be stored while keeping all the essential information intact. Furthermore, other desirable features and characteristics of the present invention will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical field and background.